Development and Validation of a Scoring Rubric for Editorial Evaluation of Peer-review Quality: A Pilot Study.

Introduction: Despite the importance of peer review to publications, there is no generally accepted approach for editorial evaluation of a peer review's value to a journal editor's decision-making. The graduate medical education editors of the Western Journal of Emergency Medicine Special Issue in Educational Research & Practice (Special Issue) developed and studied the holistic editor's scoring rubric (HESR) with the objective of assessing the quality of a review and an emphasis on the degree to which it informs a holistic appreciation for the submission under consideration. Methods: Using peer-review guidelines from several journals, the Special Issue's editors formulated the rubric as descriptions of peer reviews of varying degree of quality from the ideal to the unacceptable. Once a review was assessed by each editor using the rubric, the score was submitted to a third party for blinding purposes. We compared the performance of the new rubric to a previously used semantic differential scale instrument. Kane's validity framework guided the evaluation of the new scoring rubric around three basic assumptions: improved distribution of scores; relative consistency rather than absolute inter-rater reliability across editors; and statistical evidence that editors valued peer reviews that contributed most to their decision-making. Results: Ninety peer reviews were the subject of this study, all were assessed by two editors. Compared to the highly skewed distribution of the prior rating scale, the distribution of the new scoring rubric was bell shaped and demonstrated full use of the rubric scale. Absolute agreement between editors was low to moderate, while relative consistency between editor's rubric ratings was high. Finally, we showed that recommendations of higher rated peer reviews were more likely to concur with the editor's formal decision. Conclusion: Early evidence regarding the HESR supports the use of this instrument in determining the quality of peer reviews as well as its relative importance in informing editorial decision-making.

A Collaborative Approach to Mentored Peer Reviews Sponsored by the Council of Residency Directors in Emergency Medicine.

Introduction: Historically, there have been no systematic programs for teaching peer review, leaving trainees to learn by trial and error. Recently, a number of publications have advocated for programs where experienced reviewers mentor trainees to more efficiently acquire this knowledge. Objective: Our goal was to develop an introductory learning experience that intentionally fosters peer-review skills. Methods: The Council of Residency Directors in Emergency Medicine (CORD) offered education fellowship directors the opportunity to mentor their fellows by reviewing submitted manuscript(s) supplemented by educational material provided by their journal. Reviews were collaboratively created. The decision letter that was sent to manuscript authors was also sent to the mentees; it included all reviewers' and editor's comments, as feedback. In 2022, fellows received a post-experience survey regarding prior experiences and their perspectives of the mentored peer-review experience. Results: From 2020-2022, participation grew from 14 to 30 education fellowships, providing 76 manuscript peer reviews. The 2022 survey-response rate of 87% (20/23) revealed that fellows were inexperienced in education scholarship prior to participation: 30% had authored an education paper, and 10% had performed peer review of an education manuscript. Overall, participants were enthusiastic about the program and anxious to participate the following year. In addition, participants identified a number of benefits of the mentored experience including improved understanding of the scholarship process; informing fellows' scholarly pursuits; improved conceptualization of concepts learned elsewhere in training; and learning through exposure to scholarship. Conclusion: This program's early findings suggest that collaboration between academic societies and interested graduate medical education faculty has the potential to formalize the process of learning peer review, benefitting all involved stakeholders.

Mental Practice: Applying Successful Strategies in Sports to the Practice of Emergency Medicine.

Emergency physicians are expected to learn and maintain a large and varied set of competencies for clinical practice. These include high acuity, low occurrence procedures that may not be encountered frequently in the clinical environment and are difficult to practice with high fidelity and frequency in a simulated environment. Mental practice is a form of a cognitive walk-through that has been shown to be an effective method for improving motor and cognitive skills, with literature in sports science and emerging evidence supporting its use in medicine. In this article, we review the literature on mental practice in sports and medicine as well as the underlying neuroscientific theories that support its use. We review best-known practices and provide a framework to design and use mental imagery scripts to augment learning and maintaining the competencies necessary for physicians at all levels of training and clinical environments in the practice of emergency medicine.

An outcomes research perspective on medical education: Has anything changed in the last 18 years?

PURPOSE: Medical education research focused on patient-centered outcomes holds the promise of improved decision-making by medical educators. In 2001, Prystowsky and Bordage demonstrated that patient-centered outcomes were evaluated in fewer than one percent of studies published in a survey of major medical education journals. Though many have called for increased inclusion of patient-centered outcomes in medical education literature, it remains uncertain to what degree this need has been addressed systematically. METHODS: Using the same data sources as in the original report (Academic Medicine, Medical Education, and Teaching and Learning in Medicine), we sought to replicate Prystowsky and Bordage's study. We extracted data from original empirical research reports from these three journal sources for the years 2014-2016. We selected 652 articles that met the inclusion criteria for further analysis. RESULTS: Study participants were largely trainees (64% of studies) or faculty (25% of studies). Only 2% of studies included patients as active or passive participants. Study outcomes reported were satisfaction (40% of studies), performance (39%), professionalism (20%), and cost (1%). CONCLUSIONS: These results do not differ significantly from the original 2001 study. The medical education literature as represented in these three prominent journals has made little progress in placing a greater focus on patient-centered outcomes.

Faculty support bundle for simulation education.

BACKGROUND: Postgraduate training programmes rely on faculty to meet core educational needs, including simulation. Time is arguably the most valuable resource for academic physicians, which presents a challenge for recruiting faculty to provide extra-clinical teaching. To increase faculty engagement in simulation-based education (SBE), we first identified barriers to participation. Next, we sought to overcome barriers using a self-determination theory (SDT) framework to increase motivation using strategies that addressed faculty autonomy, competence and relatedness. METHODS: Faculty from a single department of emergency medicine were surveyed about factors influencing participation in SBE. Responses were grouped into themes and used to develop the intervention-a faculty support bundle-to overcome common barriers and promote participation. Supports focused on course materials, organisational consistency and peer recognition. Faculty participation in SBE pre- and post-implementation of the support bundle was analysed via chi-squared analysis. Faculty who delivered SBE were resurveyed after the implementation phase to explore how the support bundle affected their experience. RESULTS: Initial survey response was 41%. Reported barriers to participation in SBE included scheduling issues, preparation time, competing responsibilities, lack of confidence with simulation and lack of interest. Twenty-four faculty participated in SBE during the pre-implementation phase, compared to 39 post implementation (p = 0.03). DISCUSSION: The faculty support bundle increases faculty participation in SBE. Strategies focused on internal motivators identified using an SDT framework. In contrast to traditional external motivators, these were no cost interventions. Those seeking to increase faculty participation in SBE should consider implementing similar strategies.

A Social Network Analysis of the Western Journal of Emergency Medicine Special Issue in Educational Research and Practice.

INTRODUCTION: Scholarship and academic networking are essential for promotion and productivity. To develop education scholarship, the Council of Emergency Medicine Directors (CORD) and Clerkship Directors of Emergency Medicine (CDEM) created an annual Special Issue in Educational Research and Practice of the Western Journal of Emergency Medicine. The objective of this study was to evaluate the network created by the special Issue, and explore changes within the network over time. METHODS: Researchers used bibliometric data from Web of Science to create a social network analysis of institutions publishing in the first four years of the special issue using UCINET software. We analyzed whole-network and node-level metrics to describe variations and changes within the network. RESULTS: One hundred and three (56%) Accreditation Council for Graduate Medical Education-accredited emergency medicine programs were involved in 136 articles. The majority of institutions published in one or two issues. Nearly 25% published in three or four issues. The network analysis demonstrated that the mean number of connections per institution increased over the four years (mean of 5.34; standard deviation [SD] 1.27). Mean degree centralization was low at 0.28 (SD 0.05). Network density was low (mean of 0.09; SD 0.01) with little change across four issues. Five institutions scored consistently high in betweenness centrality, demonstrating a role as connectors between institutions within the network and the potential to connect new members to the network. CONCLUSION: Network-wide metrics describe a consistently low-density network with decreasing degree centralization over four years. A small number of institutions within the network were persistently key players in the network. These data indicate that, aside from core institutions that publish together, the network is not widely connected. There is evidence that new institutions are coming into the network, but they are not necessarily connected to the core publishing groups. There may be opportunities to intentionally increase connections across the network and create new connections between traditionally high-performing institutions and newer members of the network. Through informal discussions with authors from high-performing institutions, there are specific behaviors that departments may use to promote education scholarship and forge these new connections.

Pediatric Airway Assessment Tool (PAAT): A Rating Tool to Assess Resident Proficiency in Simulated Pediatric Airway Skills Performance.

Introduction: The Accreditation Council for Graduate Medical Education has identified the need for assessment of core skills for pediatric and emergency medicine residents, which includes pediatric airway management. Although there are standard courses for pediatric airway management, there is no validated tool to assess basic and advanced pediatric airway skills performance. Our objective was to develop a simulation-based tool for the formative assessment of resident pediatric airway skills performance that was concise, yet comprehensive, and to evaluate the evidence supporting the argument for the tool's validity. Methods: We developed a pediatric airway assessment tool (PAAT) to assess six major domains of pediatric airway skills performance: basic airway maneuvers, airway adjuncts, bag-valve mask ventilation, advanced airway equipment preparation, direct laryngoscopy, and video laryngoscopy. This tool consisted of a 72-item pediatric airway skills assessment checklist to be used in simulation. We enrolled 12 subjects at four different training levels to participate. Assessment scores were rated by two independent expert raters. Results: The interrater agreement was high, ranging from 0.92 (adult bagging rate) to 1 (basic airway maneuvers). There was a significant trend of increasing scores with increased training level. Discussion: The PAAT demonstrated excellent interrater reliability and provided evidence of the construct's validity. Although further validation of this assessment tool is needed, these results suggest that the PAAT may eventually be useful for assessment of resident proficiency in pediatric airway skills performance.

Mental health research capacity building in sub-Saharan Africa: the African Mental Health Research Initiative.

Mental, neurological and substance use (MNS) disorders are a leading, but neglected, cause of morbidity and mortality in sub-Saharan Africa. The treatment gap for MNS is vast with only 10% of people with MNS disorders in low-income countries accessing evidence-based treatments. Reasons for this include low awareness of the burden of MNS disorders and limited evidence to support development, adaptation and implementation of effective and feasible treatments. The overall goal of the African Mental Health Research Initiative (AMARI) is to build an African-led network of MNS researchers in Ethiopia, Malawi, South Africa and Zimbabwe, who are equipped to lead high quality mental health research programs that meet the needs of their countries, and to establish a sustainable career pipeline for these researchers with an emphasis on integrating MNS research into existing programs such as HIV/AIDS. This paper describes the process leading to the development of AMARI's objectives through a theory of change workshop, successes and challenges that have been faced by the consortium in the last 4 years, and the future role that AMARI could play in further building MNS research capacity by brining on board more institutions from low- and middle-income countries with an emphasis on developing an evidence-based training curriculum and a research-driven care service.

The Inevitable Interim: Transitional Leaders in Academic Medicine.

This Invited Commentary calls attention to the growing phenomenon of interim leadership, temporary leadership during which a permanent leader is sought, in academic medicine and the gap in intentional, formal leadership training of future health care leaders. The authors consider interim leadership from the perspectives of the organization, the appointing authority, and the interim leader. The authors highlight a fundamental challenge facing interim leaders-providing stability during periods of significant change. They also stress the strategic importance of interim periods and the need for succession planning within academic medicine organizations. Drawing on personal experiences and existing literature, the authors offer a 4-stage framework for considering interim leadership at academic medicine organizations and strategies for success at each stage: (1) expectations and exploration, (2) adjusting expectations, (3) accommodation, and (4) phasing out. This Invited Commentary is intended to serve as a resource for interim leaders, those responsible for appointing them, and the organizations they are called upon to lead.

2018 Academic Emergency Medicine Consensus Conference: A Workforce Development Research Agenda for Pediatric Care in the Emergency Department.

Each year, more than 30 million children visit U.S. emergency departments (EDs). Although the number of pediatric emergency medicine specialists continues to rise, the vast majority of children are cared for in general EDs outside of children's hospitals. The diverse workforce of care providers for children must possess the knowledge, experience, skills, and systemic support necessary to deliver excellent pediatric emergency care. There is a crucial need to understand the factors that drive the professional development and support systems of this diverse workforce. Through the iterative process culminating with the 2018 Academic Emergency Medicine consensus conference, we have identified five key research themes and prioritized a specific research agenda. These themes represent critical gaps in our understanding of the development and maintenance of the pediatric emergency care workforce and allow for a prioritization of future research efforts. Only by more fully understanding the gaps in workforce needs, and the necessary steps to address these gaps, can outcomes be optimized for children in need of emergency care.

Early Impact of the Western Journal of Emergency Medicine CDEM/CORD Special Issue in Educational Research & Practice.

INTRODUCTION: In 2015, with a stated goal of disseminating best teaching practices and developing a community of educational scholars, the Council of Emergency Medicine Directors (CORD) and the Clerkship Directors of Emergency Medicine (CDEM) created an annual Special Issue in Educational Research and Practice (Special Issue) in cooperation with the Western Journal of Emergency Medicine. The intention of this study was to analyze the impact of this effort to date. METHODS: Bibliometric data was gathered on all four special issues, 2015-2019, from the Web of Science and then verified with the eScholarship website. Authorship, academic affiliation, date published, article type, and format were tabulated for descriptive analysis. Using metrics from Google Scholar, alternative scholarly impact metrics (altmetrics), and the eScholarship website, the authors identified top articles and grouped them into themes. RESULTS: Of the 136 articles included in the first four years of the Special Issue, 126 represented peer-reviewed publications with an overall acceptance rate of 25.0% (126/505). Authors from this cohort represented 103 of the 182 (56.6%) Accreditation Council for Graduate Medical Education (ACGME) programs in existence at the time of the inaugural issue. Multi-institutional studies represented 34.9% (44/126) of the peer-reviewed publications. Traditional and alternative publication metrics are reported to assess the impact of articles from the Special Issues. CONCLUSION: The Special Issue is a proven outlet to share best practices, innovations, and research related to education. Additionally, the infrastructure of this process promotes the development of individual faculty and a community of teaching scholars.

Executive Summary: The 2018 Academic Emergency Medicine Consensus Conference: Aligning the Pediatric Emergency Medicine Research Agenda to Reduce Health Outcome Gaps.

Emergency care providers share a compelling interest in developing an effective patient-centered, outcomes-based research agenda that can decrease variability in pediatric outcomes. The 2018 Academic Emergency Medicine Consensus Conference "Aligning the Pediatric Emergency Medicine Research Agenda to Reduce Health Outcome Gaps (AEMCC)" aimed to fulfill this role. This conference convened major thought leaders and stakeholders to introduce a research, scholarship, and innovation agenda for pediatric emergency care specifically to reduce health outcome gaps. Planning committee and conference participants included emergency physicians, pediatric emergency physicians, pediatricians, and researchers with expertise in research dissemination and translation, as well as comparative effectiveness, in collaboration with patients, patient and family advocates from national advocacy organizations, and trainees. Topics that were explored and deliberated through subcommittee breakout sessions led by content experts included 1) pediatric emergency medical services research, 2) pediatric emergency medicine (PEM) research network collaboration, 3) PEM education for emergency medicine providers, 4) workforce development for PEM, and 5) enhancing collaboration across emergency departments (PEM practice in non-children's hospitals). The work product of this conference is a research agenda that aims to identify areas of future research, innovation, and scholarship in PEM.

A Cognitive Apprenticeship-Based Faculty Development Intervention for Emergency Medicine Educators.

In just a few years of preparation, emergency medicine (EM) trainees must achieve expertise across the broad spectrum of skills critical to the practice of the specialty. Though education occurs in many contexts, much learning occurs on the job, caring for patients under the guidance of clinical educators. The cognitive apprenticeship framework, originally described in primary and secondary education, has been applied to workplace-based medical training. The framework includes a variety of teaching methods: scaffolding, modeling, articulation, reflection, and exploration, applied in a safe learning environment. Without understanding these methods within a theoretical framework, faculty may not apply the methods optimally. Here we describe a faculty development intervention during which participants articulate, share, and practice their own applications of cognitive-apprenticeship methods to learners in EM. We summarize themes identified by workshop participants, and provide suggestions for tailoring the application of these methods to varying levels of EM learners. The cognitive-apprenticeship framework allows for a common understanding of the methods used in clinical teaching toward independence. Clinical educators should be encouraged to reflect critically on their methods, while being offered the opportunity to share and learn from others.